Switching System with Distributed Switching Fabric

ABSTRACT

A switch encapsulates incoming information using a header, and removes the header upon egress. The header is used by both distributed ingress nodes and within a distributed core to facilitate switching. The ingress and egress elements preferably support Ethernet or other protocol providing connectionless media with a stateful connection. Preferred switches include management protocols for discovering which elements are connected, for constructing appropriate connection tables, for designating a master element, and for resolving failures and off-line conditions among the switches. Secure data protocol (SDP), port to port (PTP) protocol, and active/active protection service (AAPS) are all preferably implemented. Systems and methods contemplated herein can advantageously use Strict Ring Topology (SRT), and conf configure the topology automatically. Components of a distributed switching fabric can be geographically separated by at least one kilometer, and in some cases by over 150 kilometers.

This application is a continuation of U.S. patent application Ser. No.10/965,444 filed Oct. 12, 2004 which claims priority to provisionalapplication No. 60/511,145 filed Oct. 14, 2003; provisional applicationNo. 60/511,144 filed Oct. 14, 2003; provisional application No.60/511,143 filed Oct. 14, 2003; provisional application No. 60/511,142filed Oct. 14, 2003; provisional application No. 60/511,141 filed Oct.14, 2003; provisional application No. 60/511,140 filed Oct. 14, 2003;provisional application No. 60/511,139 filed Oct. 14, 2003; provisionalapplication No. 60/511,138 filed Oct. 14, 2003; provisional applicationNo. 60/511,021 filed Oct. 14, 2003; and provisional application No.60/563,262 filed Apr. 16, 2004, all of which are incorporated herein byreference in their entirety.

FIELD OF THE INVENTION

The field of the invention is network switches.

BACKGROUND

Modern computer networks typically communicate using discrete packets orframes of data according to predefined protocols. There are multiplesuch standards, including the ubiquitous TCP and IP standards. For allbut the simplest local topologies, networks employ intermediate nodesbetween the end-devices. Bridges, switches, and/or routers, are allexamples of intermediate nodes.

As used herein, a network switch is any intermediate device thatforwards packets between end-devices and/or other intermediate devices.Switches operate at the data link layer (layer 2) and sometimes thenetwork layer (layer 3) of the OSI Reference Model, and thereforetypically support any packet protocol. A switch has a plurality of inputand output ports. Although a typical switch has only 8, 16, or otherrelatively small number of ports, it is known to connect switchestogether to provide large numbers of inputs and outputs. Prior art FIG.1 shows a typical arrangement of switch modules into a large switch thatprovides 128 inputs and 128 outputs.

One problem with simple embodiments of the prior art design of FIG. 1 isthat failure of any given switch destroys integrity of the entireswitching system. One solution is to provide entire redundant backupsystems (external redundancy), so that a spare system can quicklyreplace functionality of a defective system. That solution, however, isoverly expensive because an entire backup must be deployed for eachworking system. The solution is also problematic in that the redundantsystem must be engaged upon failure of substantially any componentwithin the working system. Another solution is to provide redundantmodules within the system, and to deploy those modules intelligently(internal redundancy). But that solution is problematic because all thecomponents are situated locally to one another. A fire, earthquake orother catastrophe will still terminally disrupt the functionality of theentire system.

U.S. Pat. No. 6,256,546 to Beshai (March 2002) describes a protocol thatuses an adaptive packet header to simplify packet routing and increasetransfer speed among switch modules. Beshai's system is advantageousbecause it is not limited to a fixed cell length, such as the 53 bytelength of an Asynchronous Transfer Mode (ATM) system, and because itreportedly has better quality of service and higher throughput that anInternetworking Protocol (IP) switched network. The Beshai patent, isincorporated herein by reference along with all other extrinsic materialdiscussed herein

Prior art FIG. 1A depicts a system according to Beshai's '546 patent.There, pluralities of edge modules (ingress modules 110A-D and egressmodules 130A-D) are interconnected by a passive core 120. Each of theingress modules 110A-D accept data packets in multiple formats, adds astandardized header that indicates a destination for the packet, andswitches the packets to the appropriate egress modules 130A-D throughthe passive core 120. At the egress modules 130A-D the header is removedfrom the packet, and the packet is transferred to a sink in its nativeformat. The solid lines of 112A-112D depict unencapsulated informationarriving to circuit ports, ATM ports, frame relay ports, IP ports, andUTM ports. Similarly, the solid lines of 132A-D depict unencapsulatedinformation exiting to the various ports in the native format of theinformation. The dotted lines of core 120 and facing portions of theingress 110A-D and egress 130A-D modules depict information that iscontained UTM headed packets. The entire system 100 operates as a singledistributed switch, in which all switching is done at the edge (ingressand egress modules).

Despite numerous potential advantages, Beshai's solution in the '546patent has significant drawbacks. First, although the system isdescribed as a multi-service switch (with circuit ports, ATM ports,frame relay ports, IP ports, and UTM ports), there is no contemplationof using the switch as an Ethernet switch. Ethernet offers significantadvantages over other protocols, including connectionless statefulcommunication. A second drawback is that the optical core iscontemplated to be entirely passive. The routes need to be set up andtorn down before packets are switched across the core. As such Beshaidoes not propose a distributed switching fabric, he only discloses adistributed edge fabric with optical cross-connected cores. A third,related disadvantage, is that Beshai's concept only supports a singlechannel from one module to another. All of those deficiencies reducefunctionality.

Beshai publication no. 2001/0006522 (July 2001) resolves one of thedeficiencies of the '546 patent, namely the single channel limitationbetween modules. In the '522 application Beshai teaches a switchingsystem having packet-switching edge modules and channel switching coremodules. As shown in prior art FIG. 1B, traffic entering the systemthrough ports 162A is sorted at each edge module 160A-D, and switched tovarious core elements 180A-C via paths 170. The core elements switch thetraffic to other destination edge modules 180A-C, for delivery to finaldestinations. Beshai contemplates that the core elements can use channelswitching to minimize the potential wasted time in a pure TDM (timedivision mode) system, and that the entire system can use time counterco-ordination to realize harmonious reconfiguration of edge modules andcore modules.

Leaving aside the switching mechanisms between and within the coreelements, the channel switching core of the '522 application providesnothing more than virtual channels between edge devices. It does notswitch individual packets of data. Thus, even though the '522application incorporates by reference Beshai's Ser. No. 09/244824application regarding High-Capacity Packet Switch (issued as U.S. Pat.No. 6,721,271 in April 2004), the '522 application still fails to teach,suggest, or motivate one of ordinary skill to provide a fullydistributed network (edge and core) that acts as a single switch.

What is still needed is a switching system in which the switching takesplace both at the distributed edge nodes and within a distributed core,and where the entire system acts as a single switch.

SUMMARY OF THE INVENTION

The present invention provides apparatus, systems, and methods in whichthe switching takes place both at the distributed edge nodes and withina distributed core, and where the entire system acts as a single switchthrough encapsulation of information using a special header that isadded by the system upon ingress, and removed by the system upon egress.

The routing header includes as least a destination element address, andpreferably also includes a destination port address, a source elementaddress. Where the system is configured to address clusters of elements,the header also preferably includes a destination cluster address and asource cluster address.

The ingress and egress elements preferably support Ethernet or otherprotocol providing connectionless media with a stateful connection. Atleast some of the ingress and egress elements preferably have least 8input ports and 8 output ports, and communicate at a speed of at leastone, and more preferably at least 10 Gbs.

Preferred switches include management protocols for discovering whichelements are connected, for constructing appropriate connection tables,for designating a master element, and for resolving failures andoff-line conditions among the switches. Secure data protocol (SDP), portto port (PTP) protocol, and active/active protection service (AAPS) areall preferably implemented.

Systems and methods contemplated herein can advantageously use StrictRing Topology (SRT), and conf configure the topology automatically.Other topologies can be can alternatively or additionally employed.Components of a distributed switching fabric can be geographicallyseparated by at least one kilometer, and in some cases by over 150kilometers.

Various objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of preferred embodiments of the invention, along with theaccompanying drawings in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic of a prior art arrangement of switch modules thatcooperate to act as a single switch.

FIG. 1B is a schematic of a prior art arrangement of switch modulesconnected by an active core, but where the modules operate independentlyof one another.

FIG. 2 is a schematic of a true distributed fabric switching system, inwhich edge elements add or remove headers, and the core activelyswitches packets according to the headers.

FIG. 3 is a schematic of a routing header.

FIG. 4 shows a high level design of a preferred combinationIngress/Egress element

FIG. 5 shows a high level design of a preferred core element

FIG. 6 is a schematic of a Raptor™ 1010 switch.

FIG. 7 is a schematic of a Raptor™ 1808 switch.

FIG. 8 is a schematic of an exemplary distributed switching systemaccording to preferred aspects of the present invention.

FIG. 9 is a schematic of a super fabric implementation of a distributedswitching fabric.

DETAILED DESCRIPTION

In FIG. 2 a switching system 200 generally includes ingress elements210A-C, egress elements 230A-C, core switching elements 220A-C andconnector elements 240A-C. The ingress elements encapsulate incomingpackets with a routing header (see FIG. 3), and perform initialswitching. The encapsulated packets then enter the core elements forfurther switching. The intermediate elements facilitate communicationbetween core elements. The egress elements remove the header, anddeliver the packets to a sink or final destination.

Those skilled in the art will appreciate that switching (encapsulation)header must, at a bare minimum, include at least a destination elementaddress. In preferred embodiments the header also includes destinationport ID, and where elements are clustered and optional destinationcluster ID. Also optional are fields for source cluster, source element,and source port IDs. As used herein an “ID” is something that is thesame as, or can be resolved into an address. In FIG. 3 a preferredswitching header 300 generally includes a Destination Cluster ID 310, aDestination Element ID 320, a Destination Port ID 330, a Source ClusterID 340 and a Source Element ID 350. In this particular example, the eachof the fields has a length of at least 1 byte and up to 2 bytes. Thoseskilled in the art should also appreciate that the term “header” is usedhere as in a euphemistic sense to mean any additional routing data thatis included in a package that encapsulates other information. The headerneed not be located at the head end of the frame or packet.

Ingress 210A-C and egress 230A-C elements are shown in FIG. 2 asdistinct elements. In fact, they are similar in construction, and theymay be implemented as a single device. Such elements can have anysuitable number of ports, and can operate using any suitable logic.Currently preferred chips to implement the design are Broadcom's™BCM5690, BCM5670, and BCM5464S chips, according to the detailedschematics included in one or more of the priority provisionalapplications.

FIG. 4 shows a high level design of a preferred combinationingress/egress element 400, which can be utilized for any of the ingress210A-C and egress 230A-C elements. Ingress/Egress element 400 generallyincludes a logical switching frame 410, Ethernet ingress/egress ports420A-L, encapsulated packet I/O port 430, layer 2 table(s) 440, layer 3table(s) 450, and access control table(s) 460.

Ingress/egress elements are the only elements that are typicallyassigned element IDs. When packets arrive at an ingress/egress port 420,it is assumed that all ISO layer 2 fault parameters are satisfied andthe packet is correct. The destination MAC address is searched in thelayer 2 MAC table 440, where the destination element ID and destinationport ID are already stored. Once matched, the element and port IDs areplaced into the switching header, along with the destination cluster ID,and source element ID. The resulting frame is then sent out to the coreelement.

When an encapsulated frame arrives, the ID is checked to make sure thepacket is targeted to the particular element at which it arrived. Ifthere is a discrepancy, the frame is checked to determine whether it isa multicast or broadcast frame. If it is a multicast frame, the internalswitching header is stripped and the resulting packet is copied to allinterested parties (registered IGMP “Internet Group Management Protocol”joiners). If it is a broadcast frame, the RAST header is stripped, andthe resulting packet is copied to all ports except the incoming portover which the frame arrived. If the frame is a unicast frame, theelement ID is stripped off, and the packet is cut through to thecorresponding physical port.

Although ingress/egress elements could be single port, in preferredembodiments they would typically have multiple ports, including at leastone encapsulated packet port, and at least one standards based port(such as Gigabit Ethernet). Currently preferred ingress/egress elementsinclude 1 Gigabit Ethernet multi-port modules, and 10 Gigabit Ethernetsingle port modules. In other aspects of preferred embodiments, aningress/egress element may be included in the same physical device witha core element. In that case the device comprises a hybridcore-ingress/egress device. See FIGS. 6 and 7.

FIG. 5 shows a high level design of a preferred core element 500, whichcan be utilized for any of the core switching elements 220A-C. Coreelement 500 generally includes a logical switching frame 510, aplurality of ingress and/or egress ports 520A-H, one or more unicasttables 530, one or more multicast tables 540.

When an encapsulated frame arrives at an ingress side of any port in thecore element, the header is read for the destination ID. The ID is usedto cut through the frame to the specific egress side port for which theID has been registered. The unicast table contains a list of allregistered element IDs that are known to the core element. Elementsbecome registered during the MDP (Management Discovery Protocol) phaseof startup. The multicast table contains element IDs that are registeredduring the “discovery phase” of a multicast protocol's joining sequence.This is where the multicast protocol evidences an interested party, anduses these IDs to decide which ports take part in the hardware copy ofthe frames. If the element ID is not known to this core element, or theframe is designated a broadcast frame, the frame floods all egressports.

Connector elements 240A-C (depicted in FIG. 2 as RAST™, for RaptorAdaptive Switch Technology™ Header), are low level devices that allowthe core elements to communicate with other core elements over cables orfibers. They assist in enforcing protocols, but have no switchingfunctions. Examples of such elements are XAU1 over copper connectorsXAU1/XGmil over fiber connectors using MSA XFP.

FIG. 6 is a schematic of a preferred commercial embodiment of a hybridcore-ingress device, designated as a Raptor™ 1010 switch. The switch 600generally includes two 10 GBase ingress elements 610A-B, two ingresselements other than 10 GBase 615A-B, a core element 620, andintermediate connector elements 630A-D. The system is capable ofproviding 12.5 Gbps throughput.

FIG. 7 is a schematic of a preferred commercial embodiment of a hybridcore-ingress device, designated as a Raptor™ 1808 switch. The switch 700could include eight 10 GBase ingress elements 710A-D, a core element720, or eight intermediate connector elements 730A-D, or any combinationof elements up to a total of eight.

In FIG. 8 a switching system 800 includes two of the Raptor™ 1010switches 600A-B and four of the Raptor™ 1808 switches 700A-D, as well asconnecting optical or other lines 810. The lines preferably comprise a10 GB or greater backplane. In this embodiment the links between the1010 switches can be 10-40 km at present, and possibly greater lengthsin the future. The links between the core switches can be over 40 km.

Ethernet

A major advantage of the inventive subject matter is that it implementsswitching of Ethernet packets using a distributed switching fabric.Contemplated embodiments are not strictly limited to Ethernet, however.It is contemplated, for example, that an ingress element can convertSONET to Ethernet, encapsulate and route the packets as described above,and then convert back from Ethernet to SONET.

Topology

Switching systems contemplated herein can use any suitable topology.Interestingly, the distributed switch fabric contemplated herein caneven support a mixture of ring, mesh, star and bus topologies, withlooping controlled via Spanning Tree Avoidance algorithms.

The presently preferred topology, however, is a Strict Ring Topology(SRT), in which there is only one physical or logical link betweenelements. To implement SRT each source element address is checked uponingress via any physical or logical link into a core element. If thesource element address is the one that is directly connected to the coreelement, the data stream will be blocked. If the source element addressis not the one that is directly connected to this core element, thepackage will be forwarded using the normal rules. A break in the ringcan be handled in any of several known ways, including reversion to astraight bus topology, which would cause an element table update to allelements.

Management of the topology is preferably accomplished using elementmessages, which can advantageously be created and promulgated by anelement manager unit (EMU). An EMU would typically manage multiple typesof elements, including ingress/egress elements and core switchingelements.

Management Discovery Protocol

In order for a distributed switch fabric to operate, all individualelements need to discover contributing elements to the fabric. Theprocess is referred to herein as Management Discovery Protocol (MDP).MDP discovers fabric elements that contain individual management units,and decides which element become the master unit and which become thebackup units. Usually, MDP needs to be re-started in every element afterpower stabilizes, the individual management units have booted, and portconnectivity is established. The sequence of a preferred MDP operationis as follows:

Each element transmits an initial MDP establish message containing itsMAC address and user assigned priority number (if assigned 0 used if notset). Each element also listens for incoming MDP messages, containingsuch information. As each element receives the MDP messages, one of twodecisions is made. If the received MAC address is lower than the MACaddress assigned to the receiving element, the message is forwarded toall active links with the original MAC address, the link number it wasreceived on, and the MAC address of the system that is forwarding themessage. If a priority is set, the lowest priority (greater than 0) isdeemed as lowest MAC address and processed as such. If on the other handthe received MAC address is higher than the MAC address assigned to thereceiving element, then the message is not forwarded. If a priority isset that is higher than the received priority, the same process iscarried out

Eventually the system identifies the MAC address of the master unit, andcreates a connection matrix based on the MAC addresses of the elementsdiscovered, the active port numbers, and the MAC addresses of each ofthe elements, as well as each of their ports. This matrix is distributedto all elements, and forms the base of the distributed switch fabric.The matrix can be any reasonable size, including the presently preferredsupport for a total of 1024 elements.

As each new element joins an established cluster, it issues a MDPinitialization message, which is answered by a stored copy of theadjacency table. The new element insert its own information into thetable, and issues an update element message to the master, which in turnwill check the changes and issue an element update message to allelements.

Heart Beat Protocol

Heart Beat Protocol enables the detection of a faked element. If anelement fails or is removed from the matrix, a Heart Beat Protocol (HBP)can be used to signal that a particular link to an element is not inservice. Whatever system is running the HBP sends an element updatemessage to the master, which then reformats the table, and issues anelement update message to all elements.

It is also possible that various pieces of hardware will send aninterrupt or trap to the manager, which will trigger an element updatemessage before HBP can discover the failure. Failure likely to bedetected early on by hardware include; loss of signal on opticalinterfaces; loss of connectivity on copper interfaces; hardware failureof interface chips. A user selected interface disable command orshutdown command can also be used to trigger an element update message.

Traffic Load

Traffic Load factors can be calculated in any suitable manner. Incurrently preferred systems and methods, traffic load is calculated bylocal management units and periodically communicated in element loadmessages to the master. It is contemplated that such information can beused to load balance multiple physical or logical links betweenelements.

Security

Element messages are preferably sent using a secure data protocol (SDP),which performs an ACK/NAK function on all messages to ensure theirdelivery. SDP is preferably operated as a layer 2 secure data protocolthat also includes the ability to encrypt element messages betweenelements.

As discussed elsewhere herein, element messages and SDP can also be usedto communicate other data between elements, and thereby support desiredmanagement features. Among other things, element messages can be used tosupport Port To Port Protocol (PTPP), which provides a soft permanentvirtual connection to exist between element/port pairs. As currentlycontemplated, PTPP is simply an element-to-element message that setsdefault encapsulation to a specific element address/port address forsource and destination. PTPP is thus similar to Multiprotocol LabelSwitching (MPLS) in that it creates a substitute virtual circuit. Butunlike MPLS, if a failure occurs, it is the “local” element thatautomatically re-routes data around the problem. Implemented in thismanner, PTPP allows for extremely convenient routing around failures,provided that another link is available at both the originating(ingress) side and the terminating (egress) side, and there is no otherblockage in the intervening links (security/Access Control List(ACL)/Quality of Service (QoS), etc),

It is also possible to provide a lossless failover system that will notlose a single packet of data in case of a link failure. Such a systemcan be implemented using Active/Active Protection Service (AAPS), inwhich the same data is sent in a parallel fashion. The method isanalogous to multicasting in that the hardware copies data from themaster link to the secondary link. Ideally, the receiving end of theAAPS will only forward the first copy of any data received (correctly)to the end node.

Super Fabric

Large numbers of elements can advantageously be mapped together inlogical clusters, and addressed by including destination and sourcecluster IDs in the switching headers. In one sense, cluster enabledelements are simply normal elements, but with one or more links that arecapable of adding/subtracting cluster address numbers. A system thatutilizes clusters in this manner is referred to herein as a superfabric. Super fabrics can be designed to any reasonable size, includingespecially a current version of super fabric that allows up to 255clusters of 1024 elements to be connected in a “single” switch system.

As currently contemplated, the management unit operating in super fabricmode retains details about all clusters, but does not MAC address data.Inter-cluster communication is via dynamic Virtual LAN (VLAN) tunnelswhich are created when a cluster level ACL detects a matched sequencethat has been predefined. Currently contemplated matches include any of:(a) a MAC address or MAC address pairs; (b) VLAN ID pairs; (c) IP subnetor subnet pair; (d) TCP/UDP Protocol numbers or pairs, ranges etc; (e)protocol number(s); and (f) layer 2-7 match of specific data. Themanagement unit can also keep a list of recent broadcasts, and perform amatching operation on broadcasts received. Forwarding of previously sentbroadcasts can thereby be prevented, so that after a learning periodonly new broadcasts will forwarded to other links.

Although clusters are managed by a management unit, they can continue tooperate upon failure of the master. If the master management unit fails,a new master is selected and the cluster continues to operate. Inpreferred embodiments, any switch unit can be the master unit. In caseswhere only the previous management has failed, the ingress/egresselements and core element are manageable by the new master over aninband connection.

Inter-cluster communication is preferably via a strict PTPP based matrixof link addresses. When a link exists between elements that receivedencapsulated packets, MDP discovers this link, HBP checks the link forhealth, and SDP allows communication between management elements to keepthe cluster informed of any changes. If all of the above is properlyimplemented, a cluster of switch elements can act as a single logicalGigabit Ethernet or 10 Gigabit Ethernet LAN switch, with all standardsbased switch functions available over the entire logical switch.

The above-described clustering is advantageous in several ways.

-   -   Link Aggregation IEEE 802.3ad can operate across the entire        cluster. This allows other vendors' systems that use IEEE        802.3ad to aggregate traffic over multiple hardware platforms,        and provides greater levels of redundancy than heretofore        possible.    -   Virtual LANs (VLANs) 802.1Q can operate over the entire cluster        without the need for VLAN trunks or VLAN tagging on inter-switch        links. Still further, port mirroring (a defacto standard) is        readily implemented, providing mirroring of any port in a        cluster to any other port in the cluster.    -   Pause frames received on any ingress/egress port can be        reflected over the cluster to all ports contributing to the        traffic flow on that port, and pause frames can be issued on        those contributing ports to avoid bottlenecks.    -   ISO Layer 3 (IP routing) operates over the entire cluster as        though it was a single routed hop, even though the cluster may        be geographically separated by 160 Km or more.    -   ISO Layer 4 ACLs can be assigned to any switch element in the        cluster just as they would be in any standard layer 2/3/4        switch, and a single ACL may be applied to the entire cluster in        a single command.    -   IEEE 802.1X operates over the entire cluster, which would not        the case if a standard set of switching systems were connected.

In FIG. 9, a super fabric implementation 900 of a distributed switchingfabric generally includes four 20 Gbps pipes 910A-D, each of which isconnected to a corresponding cluster 920A-D that includes a controlelement 922A-D that understand the cluster messaging structure. Withineach cluster there are numerous ingress/egress elements 400 coupledtogether. In this particular embodiment there each of the controlelements 922A-D has two 10 Gbps pipes that connect the ingress/egresselements 400 for intra-cluster communication. There are alsointer-cluster pipes 930A-D, which in this instance also communicate at10 Gbps.

Thus, specific embodiments and applications of distributed switchingfabric switches have been disclosed. It should be apparent, however, tothose skilled in the art that many more modifications besides thosealready described are possible without departing from the inventiveconcepts herein. The inventive subject matter, therefore, is not to berestricted except in the spirit of the appended claims. Moreover, ininterpreting both the specification and the claims, all terms should beinterpreted in the broadest possible manner consistent with the context.In particular, the terms “comprises” and “comprising” should beinterpreted as referring to elements, components, or steps in anon-exclusive manner, indicating that the referenced elements,components, or steps may be present, or utilized, or combined with otherelements, components, or steps that are not expressly referenced.

1. A system comprising: first and second networked data switches, eachhaving elements that interconnect at layer 2 of the OSI model; theswitches capable of communicating with each other using Ethernet, andoperatively coupled with each other by optic fiber over a distance of atleast 10 Km.
 2. The system of claim 1, wherein each of the switches hasa XAUI interface.
 3. The system of claim 1, further comprising aconverter that converts XAUI to XFP.
 4. The system of claim 3, whereinthe converter (phy) includes four lanes, each of which operates at arate of at least 3.125 Gbps.
 5. The system of claim 1, wherein theswitches are situated at a distance of at least 40 Km from one another.6. The system of claim 1, wherein the switches situated at a distance ofat least 150 Km from one another.
 7. The system of claim 1, wherein theswitches execute protocols that establish a stateful connection betweenthe switches.
 8. The system of claim 1, wherein each of the switchesfurther includes a copper connection.